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ABSTRACT 

Meta-analysis, a way of cumulating the results of 
research studies, focuses on the magnitudes of effect sizes and seeks 
to explain effects through study characteristics. This meta-analysis 
used the methods developed by G. V. Glass to summarize the research 
on mail survey response rate manipulations. A narrative review using 
the same studies set the stage for this review and a related review 
using descriptive methods . Resul ts will demonstrate the similarities 
and differences one may see in analyzing the same data in different 
ways, as well as indicating the ef f ect iveness of treatments meant to 
enhance mail survey response rates. Studies were coded and repeated 
measure main effects were summarized, as were absolute mean 
differences between the two levels of the variables and the 957o 
absolute confidence intervals for the absolute differences. Results 
were clearcut in that treatments were either strong or far from 
significant. Primary findings suggest that class of postage, 
incentives (especially when enclosed), followup, precontact, 
personalization of the cover letter, personalization of postage, and 
sponsorship to be effective methods of inducing responses to mail 
surveys. Questionnaire length also has an effect on response rate. 
Found to be ineffective were use of anonymity, color, cover letter 
appeals, deadlines, personalization of labels, and the status of the 
sender. Appendixes give category codings and correlations among 
dependent variables. (Contains 10 tables and 19 references.) (SLD) 
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Introduction 




smal! as seven, a narrative review and a quantitative review 



is. Cooper and 
studies is as 
will often reach 



different conclusions. 



numerical estimates than narrative review, and has Wn shown <-n ho 





survey return rates for independent and nonindependent studies and found 
little difference between the two analyses. The effects of nonindependence 
may be more severe when the ratio of independent to nonindependent studies is 
low. Second, coding of independent variables often results in loss of 
information about qualitative differences in those variables, reducing the 
outcome of the analysis to gross rather than fine-grained statements. Third 
these methods capitalize on chance, with potentially deceptive regression 
weights obtained when the number of studies is small. 

Psychometric meta-analysis (Hunter & Schmidt, 1990) also focuses on 
effect sizes but corrects them for measurement error and other artifacts 
providing an interval around an estimated true effect size. Use of this’ 
method presumes one has information such as instrument reliability and range 
restrictions. The psychometric meta-analyses of Hunter and Schmidt are an 
extension of Glass' methods rather than an incompatible alternative. 

The present study used Glass' methods to empirically summarize the 
research on mail survey response rate manipulations. Boser and Clark (1996) 
conducted a narrative review using the same set of studies. The purpose of 
this presentation is two-fold. First, the results obtained speak to the 
simil ari ti es and differences one may see when analyzing common data with two 
different methods. Second, results provide a synthesis of literature 
regarding the effectiveness and strength of treatments used to enhance mail 
survey response rates as well as the predictive strength of selected 
characteristics of the survey setting. 
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Reviews of the Response Rate Literature 



Early reviews of the effectiveness of treatments used to enhance mail 
survey response rates were narrative. This method was joined by regression 
approaches and then meta-analyses as ways of providing quantitative indices of 
the size of treatment effects and the significance of variables reflective of 
the survey setting. The results of selected recent reviews are briefly 
summarized here to provide a context for interpretation of the results of the 
present meta-analysis. For a more complete review of the reviews, consult 
Boser and Clark (1993) and Rodgers and Worthen (1995). 

Anonymity . Jobber (1986) suggests anonymity is preferable to 
identification of survey respondents, due to a slight average improvement in 
response rates but also due to the potential for more accurate reporting of 
sensitive information. Yammarino, Skinner, and Childers (1991) found no 
significant effect for anonymity. 

Color. Color of the survey paper has been found to have no substantial 
effect on response rates (Jobber, 1986), though a significant effect was found 
by Fox, Crask, and Kim (1988) with an average gain of 2% in responses when 
using green paper rather than white paper . Differences between treatment 
conditions for the 10 observations obtained from three studies ranged from -6% 
to +9%. 

Cover Letter Appeals . Jobber (1986) found this treatment to have 
nonsignificant effects, though with a slight edge for a "help-the-sponsor" 
approach over social utility or egoistic appeals. Yammarino, Skinner, and 
Childers (1991) found significant response rate differences due to type of 
appeal, but did not define what they meant by an appeal. They found no 
effects of survey setting variables on the effect of appeals. 

Deadlines . Use of deadlines has been supported due to their effects on 
response speed not on response rates (Jobber, 1986). Fox, Crask, and Kim 
(1988) found a mean effect of 1.7% of notifying survey recipients of a cutoff 
date. Study effects ranged from -14% to +8%. They identified 12 observations 
from a sample of 5 studies. 

Follow-up . Rodgers and Worthen (1995) found an average gain of 14% 
across all forms of follow-up (letter, postcard, telephone) based on 15 
observations. Differences between follow-up and no follow-up conditions 
ranged from 10% to 20%. Bruvold and Comer (1988) found follow-up contacts to 
have a stronger effect on responses than advance contacts. 

Incentives . Two reviews of the effects of incentives on response rates 
were conducted using meta-analysis, one using Glassian methods on 85 
observations (Hopkins & Gullickson, 1992) and the second using psychometric 
meta-analysis on 74 observations (Church, 1993). Hopkins and Gullickson found 
an average increase of 19% (95% Cl: 17%, 22%) with enclosed incentives and an 

average 7% (95% Cl: 3%, 12%) with promised incentives. Church found a 24% 
average increase for enclosed incentives. Both studies found a significant 
relationship between amount of the enclosed monetary gratuity and response 
rate. Neither review found any effect for context variables (population 
description, questionnaire length, and year of publication). 

Person alization . Personalization manipulations may be applied to the 
cover letter and survey, to the mailing labels, or in the form of stamped 
envelopes versus metered envelopes; Jobber (1986) suggests personalization of 
the cover letter/survey to wield no power over response rates, and cautions 
that personalization may be counterproductive when sensitive information is 
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desired. In contrast, Rodgers and Worthen (1995) found a small (6.5%) effect 
of personalized versus generic cover letters. 

Armstrong and Lusk (1987) located 6 studies differencing response rates 
for metered versus stamped envelopes. Stamps included regular, commemorative, 
and multiple small-denomination stamps. An average gain of 3% was found with’ 
use of greater personalization of postage, which was nonsignificant. 

postage Class. Armstrong and Lusk (1987) reviewed 34 studies of the 
effects of classes of return postage on response rates. In 20 
studies listing differences in response rates between first class postage and 
business reply mail, an average gain of 9% (95% Cl: 7%, 11%) was found. This 

is higher than the average gain reported by Fox, Crask, and Kim (1988) of 6%, 
with differences ranging from -10% to +32%. 

Precontact . Effects of precontact on response rates have been reviewed 
by Schlegelmilch and Diamantopoulos (1991), Haggett and Mitchell (1994), and 
Rodgers and Worthen (1995), among others. Schlegelmilch and Diamantopoulos 
examined 27 publications with 73 observations, Haggett and Mitchell found 11 
studies for 15 observations dealing strictly with industrial populations, and 
Rodgers and Worthen based their analysis on 25 observations. Schlegelmilch 
and Diamantopoulos found an average gain of 13% (differences ranged from -16% 
to +51%) over all forms of precontact, with an average 8% (differences ranged 
from -16% to +26%) for mail precontact. Haggett and Mitchell found an average 
gain of about 6% (from -7 to +16%) over mostly letter precontact. Rodgers and 
Worthen reported an average gain of 10% over all forms of precontact. 

.Questionnaire Length. Yammarino, Skinner, and Childers (1991) found 
response rates to questionnaires longer than four pages to be lower than those 
to shorter questionnaires. Heberlein and Baumgartner (1978), in contrast, 
found no significant effect of questionnaire length on response rates. 

Sponsorship . Fox, Crask, and Kim (1988) found university sponsorship to 
be a strong response facilitator, with an average gain of 9% (study range from 
-11% to +23%). Yammarino, Skinner, and Childers (1991) also suggest a gain, 
but found it minimal. 

When numerous studies exist in a field, and several attempts have been 
made to integrate their findings, it may be inevitable that conflicting 
results arise. Reviews of treatments affecting response rates have differed 
in variable definition, in study location and inclusion criteria, in method of 
study integration, and in complexity of design (i.e., simple effects of 
treatment versus interactive or confounded effects). The reviews have come to 
differing conclusions regarding the effect size of treatments designed to 
affect response rates. For example, Fox, Crask, and Kim (1988) found 
university sponsorship to be highly effective while Yammarino, Skinner, and 
Childers (1991) found it minimally effective. There is also disagreement 
about the confounding effects of variables describing the survey setting on 
response rates. Significant effects of setting variables have been found with 
regression approaches to study integration but less frequently with Glassian 
meta-analytic techniques. There is more consistency regarding whether major 
response facilitators such as incentives and follow-up are significant or not. 
A contribution of the present study is the opportunity it affords the reader 
to reflect on the differences in results found when the same set of studies 
were approached using two different integration methods. 

Method 

Appropriate articles were identified, selected, information abstracted, 
and the data coded. Article location criteria have been listed in the first 
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symposium paper as have article deletion criteria. We now move to a review of 
the criteria for including studies within articles in the analysis and then to 
the categories and procedures used to abstract and code data in this study. 

Criteria for Inclusion 

While a common set of articles was the starting point for this meta- 
analysis and the narrative review, both studies within articles, articles and 
treatment variables were eliminated from consideration in the meta-analysis. 
Table 1 presents a list of studies and articles that were dropped since they 
were unclassifiable. These studies/articles were single instances of 
addressing a narrowly-defined topic and utilized treatments that with even the 
most liberal interpretation could not be included in a level of one of the 
defined treatment variables. Further, an arbitrary minimum of five articles 
was set for performing an analysis. Experimental treatments reported in fewer 
than five articles were excluded. As indicated in Table 1, data regarding 
salience were not analyzed nor were data regarding survey item variations or 
survey format manipulations, and we are mute on the effectiveness of those 
variables in inducing responses. 



Table 1 here 



Abstracting and Coding Studies 

Since studies were to be divided into two sets and each set reviewed by 
one of the two authors, interrater agreement in information abstracting was 
assessed. First, the variables to be abstracted were identified. They are 
listed in Table 2 along with the percentage of the sample with available 
information for that variable. Studies were dropped from the meta-analysis if 
no information was available on experimental manipulation and the 
corresponding response rates. 



Table 2 here 



Both reviewers read a common set of 20 articles and reached the levels 
of agreement presented in Table 3. Discrepancies were: 

Sample Size: Rater 1 counted nondeliverables in the sample size 

and Rater 2 did not. Decision was to subtract nondeliverables 
from total sample size. 

Random Sample: Rater 2 counted systematic samples as random while 

Rater 1 did not. Decision was to treat systematic samples as 
random. 

Cover Letter: Rater 2 counted cover letters as part of treatment 

as "no" responses and Rater 1 as "yes." Decision was to count as 
"yes" responses. 

Follow-up: Rater 2 counted follow-up as "no" if treatment and 

Rater 1 did not. Decision was to treat as "yes" if treatment. 
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Random Assignment to Treatment: Rater 2 counted as random 

assignment if cell n’s seemed equal; Rater 1 counted as random 

only if explicitly stated. Decision to count as random assignment 
only if stated. 6 

Response Rate: Disagreements were found when values had to be 

hand-calculated; i.e., when not given directly in the table. 



Table 3 here 



Disagreements were resolved for the 20-article sample. Since agreement was 
° w regarding the crucial variable, response rate, a second sample of 10 
articles was selected. Agreement rates were higher for the most critical 
varia es ( able 3) and were considered adequate to proceed with independent 

. {0ne disagreement on survey topic was due to discrepant descriptions 
within the article.) 



Coding categories were defined for primary analyses and are provided in 
Appendix A. As has been pointed out by critics of meta-analysis, category 
definitions are of necessity broad and disregardful of special case studies. 
The criticism could be leveled at the present review. For example, the offer 
of feedback was classified as a promised incentive. 



D efinition of Inde pende nt Variables, Dependent Variables, and Analyses 



The groupings of studies with sufficient numbers for analysis yielded 
fourteen independent variables. Variables are listed in Appendix A with their 
original coding. However, due to small sample sizes for some codes, 
categories were collapsed for analysis. The resulting variables and their 
levels are presented in Table 4. While thorough explication of the nature of 
the treatments may be found in the original studies or in review papers, the 
treatments will be briefly described here. 1 



Anonymity was defined as whether the respondent knew the survey 
was identifiable or not. The conditions were cued by statements 
in the cover letter or by visible or unobtrusive coding on the 
survey, for example. 

Color was manipulated simply by copying the survey on white paper 
or on colored paper. 

Cover letter appeals emphasized how the good of society would be 
bettered by provision of the requested information (a social 
utility, science, or altruistic appeal), how the survey sponsor 
would be aided by the return of the survey (help the sponsor, 
sponsor appeal), or how the respondent may have been benefited by 
return of the survey (egoistic appeal), for example, by better 
programming on television. 

Deadlines included a due date in the cover letter or merely 
requested survey return. 

Follow-ups were phone calls, letters, postcards, or second copies 
of the survey sent to nonrespondents to the original mailing. 
Incentives were enclosed tokens (e.g., tea bags, pens, money, 
checks) or promised tokens (e.g., money, copy of results of the 
survey, gift). 
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Personalization of the cover letter/survey was accomplished by 
hand-signing cover letters, personal inside addresses instead of 
Dear Ms/Sir, 'user-friendliness,' use of nicknames on the cover 
letter, computer or hand-typed instead of mimeographed letters, 
handwritten postscripts, and combination of these elements and* 
variations intended to make the respondent feel the survey was 
meant for him/her individually. Mailing labels were personalized 
if they appeared hand- typed rather than machine-generated; postage 
was personalized if, for example, regular or commemorative st am ps 
were used rather than a business reply envelope. 

Questionnaire length was assessed by number of pages or number of 
items . 

Postage class reflected mailing costs with variations being bulk 
mail, first class, air mail, special delivery, or certified mail. 
Precontact employed phone, postcard, or letter to notify the 
survey recipients that they would be receiving an important 
survey. 

Survey sponsors included universities, research firms, commercial 
organizations, or charities. Sponsorship was indicated by the 
letterhead. 

Status of the cover letter signator was varied as higher (e.g., a 
professor, CEO) or lower (e.g., a student, office manager). 



Table A here 



Five dependent variables were defined, all variations on the theme of 
differences in response rates. They were: 

- The first dependent variable was formed by treating the percentage of 
responses for the control or comparison group and the percentage of responses 
for the experimental group as a within-study variable in a repeated measures 
analysis of variance. 

- The second dependent variable, the absolute difference, was the 
difference between the response rate for one experimental condition and the 
response rate for a second experimental or comparison condition. Treatments 
such as cover letter appeals had multiple conditions with none defined as a 
control condition . 

- The third dependent variable was the relative increase in returns 
calculated as the ratio of the response rate for the experimental group to the 
response rate for the comparison group. 

- As calculation of means and standard deviations for categorical 
variables may be statistically problematic, Glass et al. (1981) suggest a 
means of translating proportions based on polytomies into a hypothetical 
metric variable. The rationale for doing so is the same as that underlying 
probit analysis (Finney, 1971). The method is to difference the standard 
normal deviates corresponding to the percentages observed in the experimental 
and comparison groups. (See Glass et al., pp. 136-138, for details.) This 
difference formed the fourth dependent variable. 

- The fifth and last dependent variable was formed by differencing the 

log-odds of the response rate for the treatment and comparison groups . Log- 
odds or logits were calculated as: ln(p/l-p). 
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Primary analyses were conducted to determine the effect of each of the 
14 treatment variables on response rate using repeated measures analysis of 
variance (ANOVA). Although ANOVA is designed for use with interval or ratio 
measures of dependent variables, Harris (1994) finds the use of ANOVA with 
proportions, feasible unless extreme proportions (close to 0.0 or 1.0) or a 
very close decision on an issue warrants a more sophisticated approach. 
Repeated measures ANOVA further requires a positive correlation between levels 

^ f J r the ^ r K Pe f ted 1 meaSUres for effective use. The correlations between groups 
defined by level of treatment, presented in Table 5, were positive and 
significant for all of the 14 independent variables. Subsequent analyses 
examined the separate influence of random assignment to treatment, topic 
(targeted or not), random sample or not, population description (general 
versus other), and journal type (business versus other) using repeated measure 
analyses of variance. Not all coded variables were selected for subsequent 
analyses due to lack of clear information for that variable or lack of 
variation in practice, e.g., surveys are most often accompanied by cover 
letters and SASEs. 



Table 5 here 



Prior to analyses, the probit and logit transformations, relative 
increase, and the absolute difference dependent variables were correlated. As 
correlations were high (Appendix B), the conceptually transparent variable of 
absolute difference (AD) was used in preference to the probit, logit, and 
relative increase transformations. The distribution of the effects was 
described using the mean difference and 95% confidence interval. Joint 
effects of survey setting characteristics were examined using multiple 
regressions with the absolute difference as the dependent variable. 

Results 

Table 6 presents a summary of the significance of the repeated measure 
main effects (response rate) across experimental and comparison groups for the 
treatment conditions. Table 6 also presents the mean absolute difference 
between the two levels of the variables and the 95% confidence interval for 
the absolute difference. For example, the mean difference in response rates 
between surveys with and without a deadline was 1.11%, with a confidence 
interval around the mean difference of -3.6% to +5.9%. Results were clearcut. 
Treatments were either strong or far from significant, with the single 
exception of questionnaire length where the effect was marginal. The 
direction of effects was consistent with expectation. For sponsorship, where 
the literature has sometimes suggested a minimal benefit of university over 
other sponsors, the present analysis suggests a clear benefit of university 
over other sponsors. Table 7 presents the mean response rate by level for 
each of the 14 treatment variables. 



Tables 6 and 7 here 



The effects of five characteristics of the survey setting on differences 
between levels of treatment in response rates in concert with each of the 14 
treatment variables were examined. Numerous combinations of survey setting 
characteristics and independent variables could not be analyzed due to the 
small sample sizes and thus cells with zero entries. Significant interactions 
between treatment and setting variables and main effects of setting variables 
are listed in Table 8. There were few relationships of any magnitude between 
setting characteristics and treatment variables. 



Table 8 



The nature of the effects found for survey setting variables are 
clarified by review of the cell means provided in Table 9. The significant 
interaction of cover letter and random assignment is suspect since one cell 
(no random assignment, sponsor appeal) had but one study representing that 
condition. The only other interaction of consequence was between follow-ups 
and the population description. Follow-up was effective with both general 
populations and professional populations but increased response rates more 
sharply when the population was professional rather than general. 



Table 9 



Multiple regressions incorporated all five survey setting 
characteristics in prediction of absolute differences in the response rates 
for the fourteen treatment conditions. As can be seen from Table 10, although 
t e multiple correlation coefficients were high in many cases, there were only 
two treatments for which the survey setting variables provided statistically 
significant prediction, cover letter appeal (social utility versus sponsor) 
and follow-up. Low statistical power resulting from small sample sizes 
explains the lack of significance. 



Table 10 



Discussion 



The primary finding of this study suggests class of postage, incentives 
(especially enclosed) , follow-up, precontact, personalization of cover 
letter/survey, personalization of postage, and sponsorship to be effective 
methods of inducing responses to mail surveys. Questionnaire length has an 
effect on response rate as well. Found to be ineffective were use of 
anonymity, color, cover letter appeals, deadlines, personalization of labels, 
and the status of the sender. Results of this meta-analysis are generally 
quite similar to those found by previous researchers. 

Previous meta-analyses of incentives (Church, 1993; Hopkins & 

Gullickson, 1992) found a higher effect for enclosed monetary incentives than 
was found in this study for any type of enclosed incentive (24% and 18% versus 



15.49%) Mean absolute differences for follow-up, precontact, 
personalization, color, deadlines, and sponsorship were all similar to results 

fn^ Lusk nZ? *£ Worthe " f 1 ? 95 '' »«. Crask aSd Kim (X988), and Armstrnig 
J 1 19 2 7 \u The mean absolute difference for questionnaire length was & 
j-ff y 1 ? 6r bban that reported by Rodgers and Worthen. The most striking 
difference in this study was a higher difference in response rates due to 8 
ciass of postage than that found by Armstrong and Lusk. Another noteworthy 
inding is agreement with Fox, Crask, and Kim regarding the effect of 

, Th ® y also found a 9% increase, in contrast with the minimal 
effect found by Yammarino, Skinner, and Childers (1991). 

offon chara £ teristics o f the survey setting had little relationship to the 
e fectiveness of treatment variables. Such null results have been noted in 

var iables yS6 Ef f per ^ 6 f whlch alS ° f ° Und little effects of contextual 

Zl * it/- E lt eCtS °i context may exist > but for most of the treatments 

detec^them 11 thlS StUdy ’ the sample of articles was too small to reliably 

Multiple dependent variables were defined in this study. 

Transformations of the absolute difference were all highly correlated a 
result consistent with that of Hopkins and Gullickson (1992). While 
transformed variables may possess preferable statistical characteristics, 
then: interpretation is clouded by their complexity. Further development of 
tne statistical analysis of proportions would be desirable. 

® asic information w f s missing from many of the articles reviewed. Over 
halt the articles had no indication of the survey length and lacked 
information about the procedure. Sampling methods were unclear and authors 
were unclear about whether the study used random assignment. A number of 
articles were discarded because response rates were not numerically presented 
Reporting standards for survey studies should be developed and adhered to 
including reporting of response rates for nonsignificant results. Provision 
of basic information would allow reviewers to eventually compile a better 
framework for understanding response rates to mail surveys. 

Care should be taken by the survey researcher in projecting the likely 
result of employing a response rate manipulation. As can be seen from the 
range of differences between treatment and comparison conditions in Table 6 
there was but one treatment (sponsorship) for which differences were positive 
in all cases. Even enclosed incentives served to depress the response rate 
for at least one unlucky survey researcher. This study describes aggregate 
effects of small numbers of studies, not natural laws. 
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Table 1 



Articles, Studi es, and Treatments Deleted from Analysis 



Author/Date 

Armstrong, 1991 
Boser, 1990a 
Brown, 1965 

Childers & Ferrell, 1979 
Dillman et al. , 1993 
Dommeyer et al., 1991 
Dommeyer, 1987 
Enger et al. , 1992 
Field, 1979 
Ferrell et al . , 1984 
Ferriss, 1951 
Ford, 1968 
Frey, 1991 

Friedman & Goldstein, 

1975 

Goldstein & Friedman, 

1975 

Goulet, 1977 
Hammond, 1959 
Jacobs, 1986 
Johnson et al . , 1993 
Jones & Lang, 1980 
Kerin & Harvey, 1976 
Kimball, 1961 
Labreque, 1978 

Martin, 1970/71 
McGinnis & Hollon, 1977 
Peterson et al . , 1989 
Pressley & Tullar, 1977 
Rucker et al., 1984 
Rudd & Maxwell, 1980 
Senf , 1987 
Sheth et al., 1980 
Sirken et al. , 1960 
Smith & Bers, 1987 
Stevens, 1974 
Swan & Epley, 1981 
Taylor et al . , 1987 

Tollefson et al., 1984 
Wagner & O'Toole, 1986 



Topic 

Type of return address 
Survey booklet format 
Screening questions 
Size of paper 
Type of survey form 
Teaser on envelope 
Negative followup appe; 
Survey format 
Gender of investigator 
Where receive survey 
SASE or not 
Q format 

Item order, cover 
Researcher ethnicity 

Cover letter form 

Follow-up letter 
Cover letter content 
Response format on Q 
Q size 
Q density 

SASE stamped vs. not 
Permit vs BR envelope 
Owner vs. manager 
sponsored 
Salience 

Where receive survey 
Salience 

Cartoons on survey 

Salience 

Salience 

Reason for no response 
Q questions 
Cover letter tone 
Number of follow-ups 
Q computer coded 
Income question 
Familiarity with 
sponsor 
Q appearance 
Use of humor 



Reason for Exclusion 
Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 
2 in set 
2 in set 

Only paper on topic 
Only paper on topic 
Only paper on topic 

Only paper on topic 

Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 

<5 in set 
2 in set 
<5 in set 

Only paper on topic 

< 5 in set 

< 5 in set 

Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 
Only paper on topic 

Only paper on topic 
Only paper on topic 



Table 2 



V ariables Abstracted and Percent of Studies with Information Avail able 
Variable 



Author 


100% 


/q unclear / unsDeciiied 


Date 


100% 




Journal 


100% 




Population Description 


99% 




Sample Size 


100% 




Use of Random Sample or Not 


80% 


20% 


Survey Topic 


82% 


18% 


Survey Length in Pages 


43% 


57% 


Survey Length in Items 


33% 


68% 


Survey Accompanied by Cover 
Letter or Not 


78% 


22% 


Survey Accompanied by SASE or Not 


50% 


50% 


Follow-Ups or Not 


97% 


3% 


-Random Assignment to Treatment 


77% 


23% 


Studv Claritv 


100% 


Researcher Defined 



Table 3 

Interrater Response Agreement 



Variable 


Sample 1 (n=20) 
% Agreement 


Sample 2 (n=10) 
% Agreement 


Population Description 


95% 


100% 


Sample Size 


65% 


88% 


Random Sample 


85% 


100% 


Survey Topic 


100% 


75% 


Survey Length 


100% 


100% 


Survey Accompanied by 


90% 


88% 


Cover Letter 


Survey Accompanied by 


90% 


88% 


SASE 


Follow-ups Used 


90% 


100% 


Random Assignment to 


85% 


100% 


Treatment 


Treatments 


100% 


100% 


Response Rates 


80% 


100% 
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Table 4 



Independent Variables 


and Categories Used in Analyses 


Variable 


Category 1 


Category 2 Category T 


Anonymity 


Yes 


No 


Color 


Color 


White 


Cover Letter 
Appeals 


Social Utility 


Help the Sponsor Egoistic 


Deadlines 


Yes 


No 


Followups 


Yes 


No 


Incentives 


Enclosed 


Promised None 


Personalization: 






Cover Letter/Survey 


More 


Less 


Labels 


More 


Less 


Postage 


More 


Less 


Postage Class 


First Class 


Higher than First Class 


Precontact 


Yes 


(certified, special delivery) 
No 


Questionnaire Length 


Longer 


Shorter 


Sponsorship 


University 


Commercial/Research Firm 


Status of Sender 


More 


Less 


Table 5 






Correlations between Levels of Treatment 


Variable ResDonse RpI-ps 




Variable Correlations 


Anonymity 


.96*** 




Color 


.96*** 




Cover Letter Appeals 3 


.86 to .92** 




Deadlines 


.94** 




Followups 


.92*** 




Incentives 3 


.85 to .96*** 




Personalization : 






Cover Letter/Survey 


. 92*** 




Label 


, 97 *** 




Postage 


. 89*** 




Postage Class 


.78*** 




Precontact 


. 86 *** 




Questionnaire Length 


.85*** 




Sponsorship 


.83*** 




Status of Sender 


. 99 *** 





levels were calculated. 

**p <.01 ***p <.001 
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Table 6 



Effects of Treatm e nt on Survey Response Rate 
Treatment 



Anonymity 
Color (vs. White) 
Cover Letter Appeals 
Social Utility- 
Sponsor 

Sponsor-Egoistic 
Social Utility- 
Egoistic 



df 



Mean AD 



.18 

.62 

3.37 



1,12 

1,6 

1,7 



95% Cl 



.68 

.46 

11 



Deadlines 


.33 


1,6 


.59 


Followup (vs. None) 


29.59 


1.12 


.00 


Incentive : 


Enclosed, Promised, 


36.80 


2,32 


.001 


None 


Enclosed vs. None 


46.22 


1,51 


.001 


Promised vs. None 


12.47 


1,31 


.001 


Enclosed vs . 


32.91 


1.17 


.001 


Promised 

Personalization: 


Cover Letter/Survey 


15.21 


1,45 


.001 


Labels 


.34 


1,9 


.57 


Postage 


17.35 


1,27 


.001 


Postage Class 


32.28 


1,10 


.001 


Precontact 


28.33 


1,32 


.001 


Questionnaire Length 


4.91 


1,18 


.04 


Sponsorship 


21.16 


1,6 


.004 


Status of Sender 


.03 


1.6 


.87 



.70 


( -2.9, 


4.3) 


1.26 


(-2.6, 


5.2) 


-2.25 


(-7.1, 


2.6) 


-4.14 


(-9.5, 


1.2) 


-1.98 


(-10.2, 


6.3) 


1.11 


(-3.6, 


5.9) 


12.30 


( 7.4, 


17.2) 


10.77 


( 8.7, 


12.8) 


15.49 


(12.8, 


18.2) 


3.00 


( 1-3, 


4.7) 


15.60 


( 9.9, 


21.3) 


3.68 


( 1.8, 


5.6) 


.95 


(-2.7, 


4.6) 


5.72 


( 2.9, 


8.5) 


16.08 


( 9.8, 


22.4) 


10.40 


( 6.6, 


14.2) 


4.69 


( -2, 


9.1) 


9.45 


( 4.4, 


14.5) 


.16 


(-2.0. 


2.3) 



Range of AD 



-7.2 

-6.4 



to 13.4 
to 6.9 



-17.5 to 4.7 



-14.0 

-21.4 

-5.8 

-.9 



to 3.0 
to 11.0 

to 7.6 
to 27.3 



-7.0 to 32.5 



-7.0 

-6.5 

-5.2 



-15.5 

-7.0 

-3.8 

-3.0 

- 11.0 

-11.9 

1.9 

-3.0 



to 38.4 
to 10.5 
to 44.9 



to 15.0 
to 11.0 
to 31.0 
to 27.5 
to 37.0 
to 22.0 
to 19.1 
to 3.2 



Note: 

, — ^ 1U1 LicaLuiciiL Uid-L li tueCLb 

associated df and p. value; AD = absolute difference in response rates 
treatment and comparison groups. 



with 

between 



Table 7 



R_esponse Rate s for Treatment Variables by Level 



Treatment 

Anonymity 


Level 1 
Yes 


Mean 

52% 


SD 

21% 


Color 


Color 


33% 


15% 


Cover Letter Appeals 


Social 


40% 


12% 


Deadlines 


Yes 


35% 


10% 


Followup 


Yes 


46% 


20% 


Incentive 

Personalization: 


Enclose 


49% 


18% 


Cover Letter, Survey 


More 


41% 


17% 


Labels 


More 


40% 


20% 


Postage 


More 


39% 


16% 


Postage Class 


First 


27% 


12% 


Precontact 


Yes 


48% 


17% 


Questionnaire Length 


Longer 


45% 


17% 


Sponsorship 


Univ 


40% 


8% 


Status of Sienator 


Higher 


47% 


18% 



Level 2 Mean SD Level 3 Mean SD 



No 


51% 


21% 








White 


32% 


15% 








Sponsor 


40% 


14% 


Egoistic 


43% 


17% 


No 


34% 


13% 








No 


35% 


14% 








Promise 


33% 


19% 


None 


32% 


18% 



Less 


38% 


16% 


Less 


39% 


21% 


Less 


33% 


13% 


Higher 


43% 


15% 


No 


38% 


22% 


Shorter 


50% 


17% 


Other 


30% 


10% 


Lower 


47% 


18% 



Table 8 



Effects of Characteristics 


of the Survev Setting 


on Rpsnonsp Rsf-oc 












Treatment Variable 


Survey Setting 


F 


df 


n 


Cover Letter Appeal 
Social Utility vs. 
Help the Sponsor 
Interaction 3 
Followup (vs. None) 


RA to Treatment 


14.16 


1,8 


.006 


Main Effect 


RA to Treatment 


6.50 


1,11 


.03 


Interaction 


Topic 


6.06 


1,11 


.04 


Interaction 
Incentive : 


Population 


11.47 


1,11 


.006 


Enclosed vs. Promised 
Interaction 


Population 


4.30 


1,49 


.05 


Enclosed vs. None 
Main Effect 
Personalization: 

Cover Letter, Survey 


Population 


4.95 


1,16 


.05 


Interaction 
Postage Class 


Journal 


5.22 


1,44 


.03 


Interaction 


Random Samole 


6.38 


1,9 


.04 



levels of treatment. 

Two of four cells with n = 1. 
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Table 9 



Ce ll Mean Response Rates fo r Main and Interactive Effects of Setting VaH«Mes 



Effect 


Setting Variable 


Treatment 
Level 1 


Variable 
Level 2 






Cover Letter Appeals 


Interaction 


Random Assignment: 


Social 


Sponsor 




Yes 


39% 


47% 




No 


39% 


64% 






Follow-UD 




Interaction 


Topic: 


Yes 


No 




Targeted 


56% 


38% 




Nontargeted 


42% 


33% 






Follow-UD 




Interaction 


Population: 


Yes 


No 




General 


41% 


34% 




Other 


54% 


36% 






Postage 




Interaction 


Random Sample: 


First 


Higher 




Yes 


33% 


40% 




No 


25% 


45% 






Personalization (Letter, Survevi 


Interaction 


Journal : 


More 


Less 




Business 


39% 


37% 




Other 


44% 


38% 






Incentive 




Interaction 


Population: 


Enclosed 


Promised 




General 


38% 


23% 




Other 


57% 


42% 






Follow-UD 




Main Effect 


Random Assignment: 








Yes 


36% 






No 


58% 








Enclosed/Promised Incentives 


Main Effect 


Population: 








General 


36% 






Other 


45% 
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Table 10 



. Regression of Response Rate o n Treatment and Survey Setting Characteristics 



Treatment 


R 


F 


D 


Significant 

Predictors 


Anonymity 


.61 


.83 


.56 




Color 


.96 


5.53 


.16 




Cover Letter Appeals: 


Social Utility - Sponsor 


.97 


13.33 


.02 


RA, RS , Journal 


Sponsor - Egoistic 


.81 


1.46 


.39 




Social Utility - Egoistic 


.95 


7.73 


.07 




Deadlines 


.63 


.64 


.64 




Followup 


.88 


4.97 


.m 




Incentive: 


Incentive vs. None 


.22 


.64 


. 67 




Enclosed vs. None 


.30 


.88 


. 50 




Promised vs. None 


.23 


.28 


.92 




Enclosed vs. Promised 
Personalization : 


.64 


1.65 


.22 




Cover Letter, Survey 


.48 


2.37 


.06 




Labels 


.91 


2.50 


.41 




Postage 


.35 


.63 


.68 




Postage Class 


.90 


4.49 - 


.06 




Precontact 


.31 


.58 


.71 




Questionnaire Length 


.39 


.47 


.79 




Sponsorship 


.91 


1.54 


.52 




Status of Sender 


.71 


1.05 


.49 





Note: RA - random assignment to treatment or not; RS = randomly selected 

sample or not; Journal = business journal or journal from other discipline 
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APPENDIX A 



Category Coding 



Survey Context Variahloc 
Author: 

Date : 

Journal : 

Population Description: 



Sample Size 
Random Sample: 

Topic: 

Survey Length in Pages: 
Survey Length in Items: 
Cover Letter: 

SASE: 

Follow-Up : 

Random Assignment to 
Treatment : 

Clarity: 

Treatments 
Incentives : 

Precontact : 

Personalization of cover 
letter or survey: 

Labels personalized: 
Postage personalized: 

Status of sender: 

student) 

Questionnaire length: 
Appeals: 



Sponsor: 
Deadline : 
Anonymity: 

Type of postage: 
Follow-up: 

Color : 

Salience : 



l=business 2=education 3=psych 4=other 
l=phone directory or other general public 
2=profesional except teachers 3=college 
students/teachers 4=military 5=other 
6=no information 

0=no l=yes 2=unclear 3=nonrespondents 

0=not specified l=targeted 2=not targeted 
88=no info 99=treatment 
26-28 888=no info 999=treatment 

0=no l=yes 2=unclear 3=treatment 

0=no l=yes 2=unclear 3=treatment 

0=no l=yes 2=unclear 3=treatment 
0=no l=yes 2=unclear 

l=yes 2=moderate 3=no 



Enclosed Promised None 
Yes No 
More Less 

More Less 

More (e.g., stamps) Less (e.g., metered, 
busines reply) 

More (e.g., professor) Less (e.g., grad 
Longer Shorter 

Social Utility Help the Sponsor, Need Reply 
Egoistic, Benefit to Receiver 

University Commerical Firm Research Firm 
Yes No 
Yes No 

First class Better than first class 
Phone Letter Postcard Other No 
Color White 
More Less 



19 




21 



APPENDIX B 

Correlations Among Dependent Variables 



Variable 


AD 


Probit 


Anonymity 


Probit 


.99 




Logit 


.99 


.99 


Relative Increase 


-.90 


-.94 


Color 


Probit 


.95 




Logit 


.99 


.96 


Relative Increase 


.94 


.94 


Cover Letter Appeals 3 


Probit 


.99 




Logit 


.99 


.99 


Deadlines 


Probit 


.99 




Logit 


.99 


.99 


Relative Increase 


.96 


.98 


Follow-up 


Probit 


.92 




Logit 


.97 


.96 


Relative Increase 


.71 


.78 


Incentives 


Probit 


.90 




Logit 


.88 


.97 


Relative Increase 


.60 


.81 


Personalization of: 
Cover Letter/Survey 


Probit 


.99 




Logit 


.98 


.99 


Relative Increase 


.87 


.98 


Label® 


Probit 


.99 




Logit 


.98 


.99 


Postage 


Probit 


.99 




Logit 


.98 


.99 


Relative Increase 


.80 


.83 


Postage Class 


Probit 


-.95 




Logit 


.93 


-.99 


Relative Increase 


.66 


-.85 


Precontact 


Probit 


.98 




Logit 


.98 


.99 


Relative Increase 


.93 


.96 


Questionnaire Length® 


Probit 


-.99 




Logit 


-.99 


.99 




Logit 

-.94 



.98 



.99 



.83 



.87 



.98 



.83 



.88 



.96 
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Variable 



Sponsorship 3 

Probit 

Logit 

Status of Sender 
Probit 
Logit 

Relative Increase 



AD Probit Logit 



Note 
ff 



.87 


.99 


.92 


.99 


.99 


.99 


.97 


.99 



AD = absolute difference. 



.95 



Relative increase not computed; lacks clear control 




group 
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